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Evaluation of the reports on agency 
reviews (2005-2009) 


Executive Summary 

In accordance with its strategic plan, the Board of the European Association for Quality 
Assurance in Higher Education (ENQA) has undertaken an analysis of the first tranche 
of review reports of agencies dating from the inception of the process in 2005 until 
December 2009. The intention of this analysis is to fulfil several purposes: 

• the report can be viewed as a self-evaluation undertaken by ENQA; it captures, 
reflects on and evaluates review work carried out so far; 

• the report provides valuable information on the use and applicability of the ESG 
in agency reviews; this information will be of interest not only to ENQA and its 
members but to other E4 members and beyond, particularly when considering the 
scope and content of any future review of the ESG themselves; 

• lessons learned from the project will help to shape the second round of reviews 
which is about to start and 

• the report will prove useful in the general evolution of the respective missions of 
and methodologies used by ENQA and the European Quality Assurance Register 
for Higher Education (EQAR). 

The context within which the reviews of these agencies were carried out is important: 
quality assurance agencies were being reviewed against the principles set out in 
the ESG for the first time; the agencies are also diverse in mission, size, ethos and 
methodologies. In spite of this diversity the report evidences a significant level of 
compliance with the ESG. 

In total, 34 reports were analysed. In terms of compliance with the ESG, it is 
apparent that standards 3.2 (Official status) and 3.3 (Activities) present agencies with 
the fewest challenges. Full compliance against the other Standards was, however, 
achieved less often, specifically: 

• Standard 3.1 (Use of external quality assurance procedures): 68% full compliance; 

• Standard 3.4 (Resources): 66% full compliance; 

• Standard 3.5 (Mission statement): 66% full compliance; 

• Standard 3.6 (Independence): 61% full compliance; 

• Standard 3.7 (External quality assurance criteria and processes used by agencies): 
58% full compliance and 

• Standard 3.8 (Accountability procedures): 65%. 

These six standards have clearly presented agencies and panels with greater challenges. 
This report highlights, where possible, the extent to which any difficulties encountered 
were as a result of shortcomings within the agencies, or difficulties with the standards 
themselves, including their interpretation, or their use by panels. However, it would 
be wrong to suggest that there are wide scale problems; the report makes some 
recommendations with a view to assisting in the clearer and more consistent use of the 
standards in future. 


Those recommendations and conclusions are grouped under six headings: 

• The level of compliance and the ‘maturity’ of the agency, the legal contexts in 
which agencies work, and the year of review 

• Panels and their Judgements, including terminology and consistency 

• Problem areas - for the agencies or with the ESG? 

• Training 

• Impact of the ESG and their role in safeguarding QA in HE (have they done the 
job so far?) 

• Issues for any review of the ESG themselves. 

The report is structured to allow the reader to see the main commentary and 
conclusions first, followed by an annex with a more detailed analysis against each of the 
ESG in order to demonstrate how the conclusions were reached. A list of the reports on 
agencies that were included in the project may be found at annex 2. 


Evaluation of the reports on agency 
reviews (2005-2009): commentary 
and conclusions 


2.1 Introduction 

As part of the operation of its strategic plan, the Board of the European Association 
for Quality Assurance in Higher Education (ENQA) undertook an analysis of the first 
tranche of review reports of agencies from their inception in 2005 until December 
2009. The reviews may be carried out solely for the purpose of applying for full 
membership of ENQA or may, in addition, fulfil national requirements. It is interesting 
to note that, out of the 47 signatory countries of the Bologna Process, only 24 have an 
agency/agencies that is/are a full member of ENQA. 

Thirty four reports were analysed. (This includes eight reports on German agencies 
which differed from the others in that only three of them make specific reference to 
the Standards and Guidelines for Quality Assurance in Higher Education in the European 
Higher Education Area (ESG). This was because, formerly, the criteria for their approval 
by the Akkreditierungsrat mirrored the ESG and the agencies had thus already 
demonstrated their level of compliance in another report). Therefore the main dataset 
against which the majority of the analysis took place is twenty six reports including one 
of the earliest reports (EVA, Denmark). The review reports are listed in Annex 2. 

This was the first time such an analysis had been carried out and the intention of 
this project is to fulfil several purposes: 

• the report can be viewed as a self-evaluation undertaken by ENQA; it captures, 
reflects on and evaluates review work carried out so far; 

• the report provides valuable information on the use and applicability of the ESG 
in agency reviews; this information will be of interest not only to ENQA and its 
members but to other E4 members and beyond, particularly when considering the 
scope and content of any future review of the ESG themselves; 

• lessons learned from the project will help to shape the second round of reviews 
which is about to start and 

• The report will prove useful in the general evolution of the respective missions of 
and methodologies used by ENQA and the European Quality Assurance Register 
for Higher Education (EQAR). 

When reading the report, it is important to bear in mind the context within which the 
reviews of these agencies were carried out. Not only were quality assurance agencies 
being reviewed against the principles set out in the ESG for the first time, but the ESG 
were also being applied by review teams to a very heterogeneous body of agencies. 
Quality assurance (QA) agencies in the European Higher Education Area (EHEA) are 
diverse in mission, size, ethos and methodologies. This diversity, arising from national 
contexts, is healthy and is one of the strengths of the EHEA. In spite of this diversity 
the report evidences a significant level of compliance with the ESG. 


2.2 Methodology 

The project was co-ordinated by three members of the ENQA Board and relied on two 
main phases of analysis: 

• A software-based, thematic analysis of all available reports, carried out by a team 
from the Quality Assurance Agency (QAA), UK; 

• A more “human” analysis of the material produced by the software carried out 
by two experts who are external to ENQA but experienced in chairing agency 
reviews and, thus, in using the ESG. 

The project also took into account the feedback letters from the ENQA Board to 
agencies following the decision on membership and the progress reports from agencies 
following its consideration of their review reports. 

The members of the project team suggested the list of themes by which the software 
would analyse the reports. This analysis was then provided to the team in advance of a 
meeting in Bath, UK. At that meeting, the thematic software material was introduced 
by members of QAA, UK and discussed by the project team. Through discussion, 
further areas for software analysis were identified and the final structure of the report 
was agreed. 

The main body of the report (i.e. the analysis of the use of the ESG) was written 
by the two external members of the team. The conclusions were agreed by the whole 
project team. 

The ENQA project team (Fiona Crozier, Josep Grifoll and Helka Kekalainen) would 
like to thank the external experts (Nick Harris and Thierry Malan) and the QAA 
Outcomes team (Harriet Barnes and Clare Morris) for all their help and support in the 
drafting of this report. Their input was invaluable and the report would not have been 
possible without their hard work and good will. The team would also like to thank 
Stephanie Maurer (OAQ, Switzerland) and Paul Mitchell (Independent, UK) for their 
assistance in reading and commenting on the report prior to its publication. 

2.3 Structure and scope of the report 

The report is structured in two parts: the first part of the report is a commentary and 
sets out the conclusions of the project team. The second part of the report provides the 
detailed analysis carried out by the external experts that led to the conclusions. The 
second part of the report is, therefore, an annex to the conclusions. 

The report starts however with a numerical analysis of the success or otherwise that 
agencies achieved in meeting the European Standards and Guidelines. These results are 
divided between Type A and Type B reviews and between nationally coordinated and 
ENQA coordinated reviews. 

As the format of review reports varies considerably, a review is only included in the 
table where a clear judgement is made in the report against an individual standard in 
Part 3. Consequently, the total number for each standard is therefore not consistent; 
neither are all reviews included in the table. 


ENQA REVIEWS OF AGENCIES: TABLE OF JUDGEMENTS FOR EUROPEAN STANDARDS AND 
GUIDELINES PART 3 



Full 

compliance 

Substantial 

compliance 

Partial 

compliance 

Non 

compliance 

3.1 Use of external quality assurance 
procedures for higher education 

15 

6 

i 

0 

Type A/Type B 

6/9 

3/3 

0/1 

- 

ENQA/National 

1/14 

4/2 

0/1 

- 

3.2 Official status 

23 

0 

1 

0 

Type A/Type B 

10/13 

- 

0/1 

- 

ENQA/National 

5/18 

- 

0/1 

- 

3.3 Activities 

23 

1 

0 

0 

Type A/Type B 

10/13 

0/1 

- 

- 

ENQA/National 

4/19 

1/0 

- 

- 

3.4 Resources 

16 

5 

3 

0 

Type A/Type B 

7/9 

3/2 

0/3 

- 

ENQA/National 

3/13 

2/3 

0/3 

- 

3.5 Mission statement 

16 

8 

0 

0 

Type A/Type B 

6/10 

4/4 

- 

- 

ENQA/National 

2/14 

3/5 

- 

- 

3.6 Independence 

16 

5 

5 

0 

Type A/Type B 

8/8 

2/3 

0/5 

- 

ENQA/National 

3/13 

1/4 

1/4 

- 

3.7 External quality assurance criteria and 
processes used by the agencies 

14 

8 

2 

0 

Type A/Type B 

6/8 

4/4 

0/2 

- 

ENQA/National 

0/14 

5/3 

0/2 

- 

3.8 Accountability procedures 

15 

7 

1 

0 

Type A/Type B 

7/8 

3/4 

0/1 

- 

ENQA/National 

3/12 

2/5 

0/1 

- 


Notes to the table: 

Type A review = review conducted for purposes of ENQA membership only 

Type B review = review conducted for national purposes as well as ENQA membership and which may include strands of Agency 
activity outside the scope of the ESG. 

ENQA = ENQA-coordinated review (applies to 7 of 34 reports in the dataset; 5 out of 7 include separate judgements for each 
standard) 

National = Nationally-coordinated review 


3. Commentary and conclusions 

The analysis of reports on agency reviews to date allows us to draw some conclusions 
which may be grouped under the following themes: 

• The level of compliance and the ‘maturity’ of agency, the legal contexts in which 
agencies work, and the year of review 

• Panels and their Judgements, including terminology and consistency 

• Problem areas - for the agencies or with the ESG? 

• Training 


• Impact of the ESG and their role in safeguarding QA in HE (have they done the 
job so far?) 

• Issues for the review of the ESG: 

- The Bologna Process 

- Public information agenda (accountability/impact of activities) 

- Enhancement/improvement (nothing in the ESG) 

- making the ESG even more effective? 

3.1 THE ‘MATURITY’ OF AGENCY, THE LEGAL CONTEXTS IN WHICH 
AGENCIES WORK, AND THE YEAR OF REVIEW IN RELATION TO LEVELS OF 
COMPLIANCE 

A number of factors might be associated with the level of compliance reported by 
evaluation panels including for example: 

• the ‘maturity’ of the agency at the time of the evaluation (years from 
establishment to ENQA report) 

• the broad legal context within which the agency works (agencies work within and 
under more or less strict legally-based frameworks and requirements) 

• the year that the evaluation was carried out (is there any evidence that agencies 
evaluated later in the period had ‘learned’ from earlier reports? Is there any 
evidence that panels were appearing to get more or less ‘soft/hard’ in reaching 
their judgements?). 

An analysis of the ‘age’ of the agency being evaluated against the numbers and areas of 
non-compliance was undertaken; the fact that changing legal or other arrangements 
mean that some agencies ‘appear’ to be quite young but are in fact based on 
organisations with a much longer existence was also taken into consideration. Perhaps 
surprisingly, this did not show any clear correlation between ‘maturity’ (apparent 
or actual) and the number of judgements other than full compliance. There was, 
however, evidence that some but not all agencies worked on the basis of developing 
methodologies in time for their need. For example, follow-up procedures were only 
under consideration/development at the time of the agency’s review because they would 
not actually be implemented for a year or two. Other agencies had, however, developed 
the full methodology from inception. Nor was there any clear evidence of ‘the older 
agencies being wiser’; i.e. that the longer established agencies were more likely to be 
fully compliant with all of the standards. 

It is difficult to generalise about the relationships between the legal contexts within 
which agencies work and their likelihood of being, or failing to be fully compliant with 
the ESG. There are two main reasons for this: 

• the legal contexts vary enormously and include: 

• single national agencies working under requirements that are more or less legally 
prescriptive (compare for example Hungary or Poland and the UK) 

• a single agency that is established within and under the legal requirements of 2 
countries (e.g. NVAO) 

• a single national agency that is also working under regional legal requirements 
(e.g. Switzerland) 

• multiple agencies working within a country on 

- a national and autonomous regional basis (e.g. Spain) 


- a ‘competitive market’ basis (e.g. Germany), 

- a basis of the differentiated HE sector (e.g. Austria) 

• the legal (and political) contexts within which agencies are working seem to be 
forever changing. 

Despite these multiple variables there seem to be no obviously consistent correlations. 
Some agencies working within rather rigid legal frameworks received few or even no 
limited compliance judgements whilst other reports quite clearly linked the basis of a 
partial compliance to the legal requirements the agency faced. It is not clear whether or 
not some panels were more or less generous in the way that they interpreted a standard 
within the context of the law to which an agency was subjected, but this remains a 
possibility. 

There is no significant change in the number of substantial/partial compliance 
judgements per report with time; panels do not appear be getting ‘softer’ or ‘harder’ 
although perhaps surprisingly agencies do not appear to be ‘learning from’ the 
comments in published reports. The latter may be merely a reflection of the fact that it 
takes agencies time to introduce changes and the period over which these reports fall 
is relatively short in comparison (it is quite likely that many SERs for the 2009 reviews 
were being prepared before publication of some of the 2008 reports). 

The coexistence of several (national/regional) agencies within a single or 
overlapping legal/political structure(s) may lead to coordination, division of labour 
and/or rationalisation issues. For example, the coexistence of national and regional or 
specialised agencies may mean that more than one set of laws must be considered; one 
agency may be dealing with two sets of laws of two countries; there may be complex 
and unequal partition of functions between national/regional/specialised agencies and 
there may be issues around recognition of the agency as the sole competent quality 
assurance organisation. 

Two aspects dominate the financial independence area: the extent to which an 
agency can be independent when being dependent on funding either directly or 
indirectly from government funds or from those it is evaluating (the latter may become 
of increasing importance if and when there is a ‘market’ in the quality assurance 
of European higher education), and the necessary resources required to undertake 
evaluations effectively. 

3.2 PANELS AND THEIR JUDGEMENTS, INCLUDING TERMINOLOGY AND 
CONSISTENCY 

Annex 1 discusses in detail the way panels reached judgements and the manner in 
which they expressed their deliberations. It would appear that panels take different 
stances when reaching a decision about the level of compliance to which an agency has 
met (or not) a standard. ‘Compliance’ is an interesting word and for some suggests only 
a binary decision - either an agency is in compliance with a standard or it isn’t. The 
introduction and use of ‘substantial compliance’ and/or ‘partial compliance’ is for some 
an oxymoron whilst for others conveniently and usefully covers the fact that agencies 
can be to a large extent in compliance, and certainly seeking to meet what might be 
regarded as ‘the spirit/essence’ of the standard, with any shortcomings being outside of 
their control - for legal, contextual or timing reasons for example. 


The reports suggest that panels have taken different approaches in reaching 
judgements about ‘compliance’. This must be of concern both in a direct sense, insofar 
as reports should be equivalent in the ways that they express their outcomes, but also 
in the use of those reports in for example applications for entry into the European 
Quality Assurance Register (EQAR). It is not within the scope of this report to reflect 
on the deliberations of EQAR but it is important that the material they are provided 
with through the ENQA/national reviews is both fair and comparable in all cases. Any 
review of the use of the ESG should consider providing greater guidance to panels on 
reaching conclusions about level of compliance’; more fundamentally it might wish to 
reconsider the use of the term ‘compliance’ and/or the membership of panels to provide 
more transparency (and practical) consistency. Greater consistency between panels 
might, for example, be achieved by having panel chairs (or secretaries with powers to 
advise) drawn from a small group of individuals who have undergone simultaneous and 
detailed induction. 

Peer review requires mechanisms for panels to be able to distinguish clearly and 
consistently between saying the equivalent of. . .“their heart is in the right place but they 
are constrained by local laws” OR “they are using local laws as a justification for what 
they do (that doesn’t really fit the ESG)” OR ‘they don’t understand what the ESG is really 
about (perhaps because they are preconditioned by their own environment and extensive 
experience to see just that approach)’. External stakeholders might suggest that for 
this important process an alternative would be to move away from peer review, but 
this could destroy a substantial part of the process; perhaps panels of peers led by a 
‘professional’ would be most effective. 

3.3 PROBLEM AREAS - FOR THE AGENCIES OR WITH THE ESG? 

The table on page X summarises the success or otherwise of agencies in meeting the 
individual standards. It is immediately apparent that standards 3.2 and 3.3 present 
agencies with the fewest challenges. Full compliance with the other standards was, 
however, achieved less often; specifically: 

• Standard 3.1 (Use of external quality assurance procedures): 68%; 

• Standard 3.4 (Resources): 66%; 

• Standard 3.5 (Mission statement): 66%; 

• Standard 3.6 (Independence): 61%; 

• Standard 3.7 (External quality assurance criteria and processes used by agencies): 
58% and Standard 3.8 (Accountability procedures): 65%. 

These six standards have clearly presented agencies and panels with greater challenges. 
The individual standards are analysed in detail in annex of this report and this 
summary conclusion seeks only to highlight, where possible, the extent to which 
any difficulties encountered were as a result of shortcomings within the agencies, or 
difficulties with the standards themselves, including their interpretation, or their use 
by panels. 

Putting aside the variations between panels and their use of the graded judgements 
it is clear that each of the ESG led to some difficulties for some panels. Most commonly, 
panels had some difficulties in applying the Independence and Accountability 
standards but there were some difficulties with others as well. However, it would be 
wrong to suggest that there are wide scale problems; attention is drawn to some 


summary points to assist in the clearer and more consistent use of the standards in 
future. 

ESG 3.1 Use of external quality assurance procedures for higher education. 

In terms of the overall assessment against Standard 3.1, one third of reports concluded 
with a judgement other than full compliance. However, the reasons for these are not 
consistent across the reports, with different agencies failing to satisfy the evaluation 
panel about different aspects of the ‘sub set’ of standards (those within ESG 2). 
Standards 2.4 (Processes fit for purpose), 2.6 (Follow-up procedures), 2.3 (Criteria for 
decisions) and 2.5 (Reporting) caused the most problems. 

One aspect that is increasingly important here is the ‘criteria for decisions’ 
particularly in terms of academic standards. With ever increasing emphasis on 
recognition of awards across borders to assist with mobility it is increasingly important 
that there are shared reference points regarding the levels of achievement attained by 
students/graduates. The proliferation of national and other qualifications frameworks 
that can and are being referenced against the Framework of Qualifications of the EHEA 
is a major advance, as is the increasing emphasis on recognition of learning outcomes. 
Reports increasingly refer to these as key components as a shift from quality assurance 
based primarily on ‘input measures’ (staff qualifications, study hours, institutional 
resources etc). One possible feature of external QA that was conspicuous by its general 
absence from reports was ‘Tuning’; despite very substantial investment by the European 
Commission it would appear from the quality assurance reports that Tuning has not 
lead to wide spread uptake in the establishment of academic standards across the range 
of disciplines. This contrasts with, for example, the impact of the European Directives 
on regulated professions and the impacts that national standards (whether legally 
required or not) have in many countries for specific subjects. 

ESG 3.2 Official status 

Whilst the standard is essential, its wording and associated guidance might be clarified 
to assist panels in making a judgement where an agency’s official status is determined 
by legal parameters that are outside of its control or even influence. 

There can be overlap between the evidence base concerning the official status and 
independence standards; could the standards (and/or their guidance) be clarified to 
reduce this or would they be better combined under a single heading? 

ESG 3.3 Activities 

The standard states that “Agencies should undertake external quality assurance activities 
(at institutional level) on a regular basis” with the guideline adding that “These may 
involve evaluation, review, audit, assessment, accreditation, or similar activities and should 
be the core functions of the agency.” It is of note that the Activities’ standard does 
not refer to any ‘improvement/enhancement’ activities although many agencies claim 
to undertake these either with regard to improvement of QA procedures themselves 
(although this aspect may be regarded as implicit in Standard 8: Accountability 
procedures) or through contributing to improvements in the ‘subject’ under evaluation 
(a programme or institution). 

The Activities standard was one that agencies under review managed to deal with 
very successfully; all undertook external quality assurance on a regular basis, although 


several reports note that the activities undertaken by the agency had changed and 
would continue to change, within structured (although sometimes quite ambitious) 
strategic and operational plans. 

Before considering the range of those activities it might be reasonable to ask ‘why, 
if it seems so straightforward for agencies, is this a standard? The answer may lie within 
the context of the stage in development of national, regional and other agencies across 
the EHEA when the ESG were first proposed. There were legitimate concerns about the 
potential risk of the development of transient, for-profit QA agencies - the evaluation/ 
accreditation equivalent of ‘degree mills’. It maybe a success of the ESG that such a 
scenario has not (yet) arisen within the ever expanding number of agencies across an 
increasingly broad EHEA. 

ESG 3.4 Resources 

Only a few SERs and reports give a detailed description of funding and expenditure, 
including their evolution in time by activities, type of expenditure and sources of 
funding, unitary costs by operation type, basic components of costs, evaluation of 
additional costs due to extra load imposed by ministries or to be forecasted because of 
expansion of missions. These examples might be used for more in-depth comparative 
questioning, template elaboration and information in a training perspective. 

A general conclusion can be summarised from the ways in which panels have applied 
this standard: 

• External quality assurance particularly at programme level can be costly, 
especially where site visits are a routine expectation but with substantial political 
backing for the Bologna Process and with quality assurance as a central pillar of 
this agencies have until now appeared to have been generally quite well resourced. 
It will be interesting to see whether this remains the case following the global 
economic downturn and it may be that agencies will need to find more cost 
efficient approaches and procedures in future. 

ESG 3.5 Mission statement 

The guidelines of ESG 3.5 (Mission statements) advise a description of the goals and 
objectives of agencies’ quality assurance processes and of the division of labour with 
relevant stakeholders in higher education, especially the higher education institutions. 
Such aspects are often set out in great detail within the official and legal status (and 
statutes) governing an agency, its structures, powers and scope of activities. There is not 
always consistency between reports and there may be overlap within reports between 
aspects covered under the ‘Official status’ and ‘Mission statement’ standards 
The standard, as currently set out, does NOT require: 

• that the goals and objectives are contained (summarised) within a single publicly 
available document 

• that panels make a judgement about whether the mission statement accurately 
reflects what the agency is actually doing. 

ESG 3.6 Independence 

The independence of QA agencies is assumed by many to be a prerequisite for ensuring 
transparently fair judgements. Many panels clearly struggled with an agency’s 


intentions for impartiality and the legal/political/funding contexts within which 
it sits where each and all could potentially have an influence. The standard states 
that “Agencies should be independent to the extent both that they have autonomous 
responsibility for their operations and that the conclusions and recommendations made in 
their reports cannot be influenced by third parties such as higher education institutions, 
ministries or other stakeholders.” 

Panel members are thus not asked to reach an assessment whether an agency is in 
itself independent but whether the agency has independent control over its operations 
and whether ‘outsiders’ can influence its conclusions etc. It appears from the texts 
that often panels stray into wider areas of independence per se. This may in part be 
because the Standard is not easily interpreted within some of the contexts in which 
agencies work. Complete ‘autonomous responsibility for their operations’ is difficult to 
imagine without, for example, complete legal and/or financial autonomy. No agency 
is in this position yet. Further the notion that an agency can reach ‘conclusions and 
make recommendations without influence of third parties such as. . . ministries and other 
stakeholders’ may be impossible to fulfil when it is such bodies that initially set the 
context and often the detailed standards in/upon which conclusions are reached and 
recommendations made. 

In some cases the conclusions reached by the agency are only, in a formal sense, 
‘recommendations’ that are made to, for example, a ministry or other body that itself 
makes the final decision. In such cases, even where there is no evidence of the latter 
body changing a decision from an agency’s recommendation, it is difficult to imagine 
that the knowledge of the working of the final body does not influence the work of an 
agency. 

For these and other reasons the reports appear less than completely consistent in the 
way that panels judge ‘compliance’ (meeting the standard as far as is reasonably possible 
within the context that the agency is working) and ‘substantial/partial compliance’ 
where a panel has not made such an allowance for the context but applied the standard 
in a stricter way. 

ESG 3.7 External quality assurance criteria and processes used by the agencies 

It is superficially surprising that only two thirds of the agencies received a (fully) 
compliant judgement for this standard when the expectations are reasonably clear 
and agencies are generally competent at procedural matters. The shortcomings were 
generally a consequence of either local ‘context’ (e.g. restrictions on public reporting all 
aspects of a review where these could lead to direct identification of individuals within 
a programme or institution) or a consequence of the age of the agency (e.g. undertaking 
follow-up procedures when the planned cycle had not yet reached that stage when the 
review for ENQA membership took place). The repeated impact of such aspects may 
give rise to consideration of two points: 

1. that in any review/revision of the ESG, it may be worth considering a greater 
emphasis on ‘meeting the intention of the standard’ with a contextual explanation 
rather than a series of associated Guidelines 

2. that the expectations within the standards might be organised in a way that better 
reflects the diverse range of contexts and experience of agencies. 


ESG 3.8 Accountability procedures 

A great deal has been written about Accountability procedures’ both by panels in 
their reports and in the annex to this report. The standard “Agencies should have in 
place procedures for their own accountability’ is vague and can appear a tautology for 
many. It is not helped by the Guidelines set out below it; agencies and the contexts 
in which they work, and panels and the contexts in which they work can have quite 
different understandings of ‘guidelines’ especially when the Guideline starts with the 
text, “These procedures are expected to include the following. . It would be useful to 
reconsider this standard and the associated guidelines in any review of the ESG. 

3.4 TRAINING 

Two aspects are considered here - training by the agencies of those undertaking their 
work and training of panel members who undertake evaluation of the agencies. 

It is clear from the reports that there is considerable variation in the extent to which 
agencies train those working for them, but there would appear to be an increasing 
recognition that this is an essentially important element within an agency’s work 
and increased efforts are being made particularly with regard to seeking improved 
consistency. Such training is not just at an academic/pedagogical level (particularly 
where programme evaluation is a large part of an agency’s work) but also at a national 
legal level since, as part of ESG 3.2 agencies “. . .should comply with any requirements 
of the legislative jurisdictions within which they operate.” These requirements are often 
laid down in great detail by legislation: in this case, short of advocating legislative 
changes, the global judgment on independence primarily has to take legislation into 
consideration. This is reflected in the variety of compliance judgments by the panels, 
and decisions can reflect different weighting of arguments, leading to sometimes mixed 
conclusions and sometimes, although more rarely, to unanswered or open questions. 

The training of members of panels evaluating agencies falls into two areas: generic 
training by ENQA and others about the expectations at a European level and to 
encourage consistency ‘across’ reviews. ENQA provides training for all panel members 
and secretaries with a view to ensuring that there is a consistent approach to the 
consideration of the ESG. All panel members are, wherever possible, drawn from 
the pool of ENQA-trained reviewers. Also, specific training may be provided by the 
co-ordinating agency at the start of each individual review. Each agency appears to pay 
proper attention to the latter and carry out this function seriously, expecting all panel 
members to be involved. 

3.5 IMPACT OF THE ESG AND THEIR ROLE IN SAFEGUARDING QUALITY 
ASSURANCE IN HIGHER EDUCATION (HAVE THEY DONE THE JOB SO FAR?) 

It is clear from the reports that the ESG have had a major impact on the quality 
assurance of higher education within the EHEA. They have provided a focus against 
which agencies can assess their own activities and standards and, importantly, 
society’s expectation of how higher education across Europe can provide reassurance/ 
accountability for what it (says it) offers. It is a trite but true statement that “quality 
assurance is (or at least should be) a journey and not a destination” but the judgements 
against the standards do provide a ‘destination’, albeit an intermediate one. There are 
clearly differences in where different agencies and the respective HE systems they are 


assessing are within a shared (but not identical) journey; it is important to recognise 
this and determine whether all should be judged to the same standard irrespective 
of their experience and ‘maturity’ or whether, and to what extent, the contextual 
background within which an agency operates can mitigate the extent of compliance 
required with the ESG. This inevitably interacts with the extent to which judgements 
may (or may not) be ‘graded’. Is ‘compliance’ the correct term when the judgement can 
be modified to one of ‘substantially’ or ‘partially’, for example? 

What a simple reading of the report does not always convey is the extent to 
which the ‘problems’ were a real cause of difficulties or perhaps an excuse for lack of 
consistent and transparent activity. Any peer assessment system will be subject to such 
variations but it is incumbent on ENQA to assist in reducing variability between panels 
by guidance and training. It might be of greater concern if a simplistic use of a panel’s 
judgements were used for subsequent actions and purposes. 

Consistency of approach and judgement is critical where ENQA review reports 
provide a significant part of the evidence used to assess whether a panel should enter 
the European Quality Assurance Register. It is interesting to note that not all agencies 
that have successfully completed an ENQA review have sought entry to EQAR; it would 
be inappropriate to discuss the possible reasons behind such decisions here but it is 
important to consider to what extent the variations between ENQA panels and their 
conclusions are reflected in the reports and might be subsequently reflected in EQAR 
decisions. Stakeholders might reasonably ask ‘how can an agency that is given a series of 
judgements other than fully compliant still gain entry to EQAR?’ 

One aspect that provoked discussions within some evaluation panels was the 
‘match’ or not between the agency’s mission statement and its activities. The ‘mission 
statement’ standard (ESG 3.5) refers to having a “. . .publicly available statement... 
translated into a clear policy and management plan”, but nothing about ‘activities today’. 
For example, it is possible for an agency to claim within the mission statement that its 
main aim and activities are to contribute to improvement/enhancement, yet in the early 
years the activities will almost certainly be concerned with building up the data and 
expertise upon which such an enhancement function might in future be delivered. In 
such a case the agency has a mission statement (compliant with 3.5) and it has regular 
activities (compliant with 3.3) but they are not (yet) directly linked. 

3.6 ISSUES FOR THE REVIEW OF THE ESG 

In their 2003 Berlin Communique, the Ministers of the signatory states of the Bologna 
Process invited the development of ‘an agreed set of standards, procedures and guidelines 
on quality assurance’ an exploration of ways ‘of ensuring an adequate peer review system 
for quality assurance and/or accreditation bodies’ and by 2005 what are now referred to 
as the ESG had been developed. In the foreword the then President of ENQA, Christian 
Thune, noted that “It must be emphasised that the report [on the ESG] is no more than a 
first step in what is likely to be a long and possibly arduous route to the establishment of 
a widely shared set of underpinning values, expectations and good practice in relation to 
quality and its assurance. . .” 

The analyses included within this report indicate the remarkable success of the ESG 
in seeking to meet those goals and within such a short period of time. The Bologna 
process has, however, developed further during the last few years and it should be 
borne in mind in any consideration of the ESG and their application in the evaluation 


of QA agencies in terms of review and, if necessary, revision. The following sections 
summarise some of these considerations. 

The Bologna Process 

The Bologna Process has transformed European HE during the last decade 
including having a major impact on quality assurance. It continues to extend its 
geographical range and its scope of priorities through the development of more 
and wider-reaching action lines. It will be crucial for the quality assurance aspects 
that they continue to focus on their core priorities and concerns of stakeholders - 
principally 1) that an HE institution is a legitimate entity, 2) that the degrees they 
award are of an agreed and appropriate standard, 3) that the degree titles reflect 
the student’s areas of knowledge, understanding and abilities, and 4) that students 
get a fair deal in so far as i) programmes actually offer what is advertised, and ii) 
students get reasonable support in their academic endeavours to assist completion 
of studies. 

It will be important that QA does not get involved in whether institutions are 
fulfilling what are essentially political aspirations regarding social engineering! 
The wider development of the Bologna Process into a political activity should not 
infiltrate into the simple and straightforward need for reporting on the QA of 
academic standards and quality. 

Public information agenda 
(accountability v. impact of activities) 

Quality assurance is increasingly linked to a ‘public information agenda’ as well 
as an accountability one. At a time when resources are being reduced it will 
be critical for independent agencies and those who direct them to determine 
the required balance of priorities between accountability, information and 
‘improvement/enhancement’. In all cases it will be essential to identify the priority 
targets/audiences as well. 

With regard to information, it is clear that different audiences need different 
things. This should not be confused, under a ‘transparency banner,’ with 
providing everything about everything and leaving everyone to sort out for 
themselves exactly what they need. An overload of data is not information and 
it should not be the QA agencies’ role to provide detailed information about 
programme content and teaching & assessment methods. It might however 
reasonably be regarded as an agency’s responsibility to determine whether the 
material an institution publishes to ‘advertise’ its offerings is accurate and an 
agency should be expected to publish its findings on this aspect. This aspect of 
becomes more important in a global market-led HE context. 

Any survey of stakeholder needs for information should be focussed on what 
is actually needed, not what might be useful or interesting! It is possible that 
what stakeholders actually need (as opposed to think they might want) and what 
agencies are geared to providing are not always matched that closely. There is 
little in the current ESG that examines whether the information provided by 
agencies is either used by stakeholders or is useful to them. Any review of the ESG 
may wish to take this into consideration. 


Enhancement/improvement 
(nothing in the ESG) 

Many reports make some comments that the ‘Activities’ standard does not refer to 
any ‘improvement /enhancement’ activities. Many agencies, through their mission 
and/or activities claim to undertake such work either with regard to improvement 
of QA procedures themselves or through contributing to improvements in the 
‘subject’ under evaluation (a programme or institution). 

To some extent the improvement of QA procedures may be regarded as 
implicit in 3.8 (Accountability procedures) but there is however ambiguity in the 
standard and certainly differences in panels’ approaches to its interpretation as to 
whether this includes improving the effectiveness and efficiency of the agency’s 
direct (external QA) work or whether it should additionally refer to improving 
the effectiveness and efficiency of institutions’/programmes’ own internal 
QA procedures. Agencies and their panels routinely make recommendations/ 
instructions for change in their reports on programmes and institutions and 
many have follow-up procedures to monitor the extent of implementation of 
the recommendations for change. There seems however to be only an implicit 
assumption that the recommendations will have actually resulted in improvement, 
without any direct assessment of impact. 

Whether or not the ESG should explicitly include reference to an assessment 
of the impact of QA on improving the quality of the educational process itself is 
an open question. This is of course something that can only be monitored over 
time - at least after one and possibly even after two ‘cycles’ of QA activity - and 
would thus only be a standard that could apply to those agencies which had been 
established for a sufficient period of time. At least two factors militate against it 
being routinely included as an ENQA standard: 

- it is only applicable to ‘older’ agencies; if compliance with the standards is 
required for ENQA membership and consideration for entry to EQAR this 
would automatically exclude any ‘young’ agencies (maturity may be prized in 
some wines and cheeses but in political terms agencies must be seen to be bona 
fide from their inception) 

- the recording and integration of monitoring results over a long period requires 
considerable resources - that are less generally available as a result of the 
current national and international economic crises 

A third ‘objection’ that might also need to be considered is that the pedagogy 
and its improvement is a specialist branch of the work of the HE sector itself 
and might be more effectively led by ‘the academy’ rather than any external QA 
function. Whilst the latter may have an important contribution to make it cannot 
be responsible/accountable for something that others more legitimately control. 

Making the ESG even more effective? 

Throughout this report there are examples of problems faced by panels because of 
the structure and detail of the way in which the ESG are written. Whilst suitable 
for some national/regional/educational contexts they are not universally easy to 
interpret and apply and thought should be given in any review of the ESG as to 
whether and, if so, how they could and should be revised. 


Such a review might consider not just the standards themselves but also 
whether the current guideline format (which seeks to exemplify what the standard 
is seeking to ‘test’) should also ask, “why/ for what purpose?” and why this is 
important. Guidance (and training) for panel members would then focus not on 
a ‘legalistic (binary) reading’ but one in which the panel brings an adequately 
detailed argument supporting its judgement and reaches a conclusion in terms of 
a ‘connoisseur judgement’. Linked to this however there will need to be greater 
emphasis on demonstrable rigour in the ways that panels are selected and work. 

3.7 CONCLUSION 

It is clear that the ESG are having an impact on the work of QA agencies and that those 
agencies are, in general, so far as they are able, complying with the standards. The 
conclusions of the report demonstrate that there are certain Standards that highlight 
areas of development for some or all agencies; a second round of reviews will be able to 
provide a value-added approach rather than simply a re-evaluation or reaccreditation. 

This part of the report has set out the conclusions of the project team. Annex 1 
provides the reader with access to more detailed information on how those conclusions 
were reached. 


Annex (1) to the report on the review 
of agencies: detailed analysis of 
the reports in terms of individual 
standards in the ESG 


Although not directly structured or arranged as such, the key standards for the 
evaluation of quality assurance agencies and the work they do is covered by three 
main and obvious strands. The detailed analysis groups the individual ESG for analysis 
purposes as follows: 

• Legitimacy of the agency to undertake evaluations of HEIs 

• Activities of the agency in its evaluation work 

• Sustainability of the agency to continue with its work. 

The ‘legitimacy’ strand is covered by the following standards in Part 3 of the ESG: 

• 3.2 Official status 

• 3.5 Mission statement 

• 3.6 Independence 

Demonstrating accountability to the interests of stakeholders may also be regarded as 
part of an agency’s ‘legitimacy,’ but because it is often more practical than the above, 
this aspect is considered primarily under ‘Activities’, but also under ‘Sustainability’. 

The ‘activities’ strand is covered by: 

• 3.1 Use of external quality assurance procedures for higher education 

which effectively involves examination of the activities under the standards of Part 2 

• 3.3 Activities 

• 3.7 External quality assurance criteria and processes used by the agencies 

• 3.8 Accountability procedures 

The ‘sustainability’ strand is covered most directly by: 

• 3.4 Resources 

and additionally some aspects of 

• 3.8 Accountability procedures 

The annex, therefore, is structured under these three main headings and the grouping 
of standards that they encapsulate. Each standard in part three of the ESG will be dealt 
with under the relevant heading. 

LEGITIMACY 

The legitimacy of agencies and their activities can be assessed through both a variety 
of legal texts and provisions as well as through a number of operational arrangements. 
Three ESG standards directly address this theme: 


3.2 Official status 

3.5 Mission statement 

3.6. Independence 

A great variety of issues and situations are analysed in the reports, but the issue of 
independence appears the most dominant and controversial in terms of assessing 
legitimacy. Some duplications, doubts or contradictions therefore appear in the 
consideration and assessment by panels of practical aspects of independence. 

ESG 3.2. Official status [ENQA criterion 2 (and Annex IV. Full membership 
provision)] 

The standard states: “Agencies should be formally recognised by competent public 
authorities in the EHEA as agencies with responsibilities for external quality assurance 
and should have an established legal basis. They should comply with any requirements of 
the legislative jurisdictions within which they operate.” 

Details regarding Official Status are often laid down in great detail by legislation. In 
this case, short of advocating legislative changes where they have reservations, panels 
generally make a judgement about whether the agency is following the spirit of the 
standard. 

Several aspects of legitimacy through official status may influence real or apparent 
independence and this interaction provides themes for discussion within the review 
panels and with stakeholder groups such as HEIs, professional organizations and 
students. These are addressed at greater length in the section on Independence since 
panels generally found the standard on Official Status less problematic. 

Some of the issues that are debated in depth by some panels reflect the diversity 
of approach to reaching a judgment relating to their areas of expertise, and the 
professional and contextual backgrounds of the panel members. Decisions often 
reflect different weighting of arguments, leading to sometimes mixed conclusions and 
sometimes, although more rarely, to unanswered or open questions. This is seen, for 
example, in: 

• occasional differentiation in the final compliance judgment, with for example 
panels concluding with comments such as: “Fully compliant - in terms of 
operations and decision making. Partly compliant - in terms of financial autonomy”; 
“Substantially compliant - in terms of operations. Partly compliant - in terms of 
decision making” 

• the formulations and terminologies used for expressing compliance decisions 
with, for example, comments such as: “in compliance with the spirit of the ENQA 
standard”; “All the remarks made about [the standard], however, do not imply non 
compliance.... [which] is sufficiently safeguarded”; “substantially but not completely 
complies 

• the level of precision in formulations and terminologies used for founding 
assertions and decisions. 

The reports included a number of comments considering the variety of contexts 
in which agencies operate, most notably and frequently with references to aspects 
of the national legal requirements and educational (and sometimes also political) 
structures, for example “. . .conforming to ESG in a broad sense with respect to the 


national peculiarity. The [national] structure requires the production of a complex balance 
between the interests of the states and regulations which affect all states. Only against the 
background of the peculiarities in the higher education system (state system with strong 
regulation, federalism), is the status quo understandable”. 

Reports also make comparative references to “the European context” (with regard to, 
for example, the pattern of funding arrangements) and thus often indicate implicitly an 
acknowledgement of the impact of the Bologna Process. 

General conclusions can be summarised from the ways in which panels have applied 
this standard: 

• whilst the standard is essential, its wording and associated guidance might be 
clarified to assist panels in making a judgement where an agency’s official status is 
determined by legal parameters that are outside of its control or even influence. 

• there can be overlap between the evidence base concerning the official status and 
independence standards; could the standards (and/or their guidance) be clarified 
to reduce this or would they be better combined under a single heading? 

ESG3.5. Mission statement. 

The standard states, “Agencies should have clear and explicit goals and objectives for their 
work, contained in a publicly available statement.” 

Mission statements are not always contained in a single statement or even a single 
document. Agencies and panels often refer to elements taken from several other legal, 
policy or organizational documents (strategic and annual operating plans). In some 
cases all elements of the mission statement are articles from very detailed laws. 

There are occasional comments on the extent to which the mission statement 
actually reflects the real functions of the agency rather than its aspirations. Many 
agencies, for example, make reference in their mission to quality improvement/ 
enhancement but it is not clear whether this refers to improvement of quality assurance 
per se or of the higher education offered. If (implicitly) it is the latter, what is the agency 
doing to promote this when it is largely committed to primary actions that may be 
regarded as quality control? 

The guidelines of ESG 3.5 advise a description of the goals and objectives of agencies’ 
quality assurance processes and of the division of labour with relevant stakeholders in 
higher education, especially the higher education institutions. Such aspects are often 
set out in great detail within the official and legal status (and statutes) governing an 
agency, its structures, powers and scope of activities. There is not always consistency 
between reports and there may be overlap within reports between aspects covered 
under the ‘Official status’ and ‘Mission statement’ standards. 

The guidelines for this standard also advise a description of the cultural and 
historical context of the agency’s work but these are also aspects that can be 
subsumed within the background to and justification of important debated aspects of 
‘Independence’. 

A general conclusion that can be summarised from the ways in which panels have 
applied this standard: 

• the standard, as currently set out, does NOT require: 

- that the goals and objectives are contained (summarised) within a single 
publically available document 


- that panels make a judgement about whether the mission statement accurately 
reflects what the agency is actually doing. 

ESG 3.6 Independence 

The standard states that, “Agencies should be independent to the extent both that 
they have autonomous responsibility for their operations and that the conclusions and 
recommendations made in their reports cannot be influenced by third parties such as 
higher education institutions, ministries or other stakeholders.” 

This topic shows much overlap between ESG 3.2 Official status and, to a lesser 
extent, ESG 3.5 Mission statement. 

In all cases, official legal documentation, in various forms, state the goals and 
rules concerning missions, functions, activities and nominations of management and 
experts. These aspects are often dealt with in great detail, notably around the issue of 
independence. 

“Independence” and “autonomy” of agencies are in most cases used as synonyms and 
are always guaranteed by legal dispositions (primary and secondary legislation, laws, 
decrees, statutes, conventions) setting out relations with Ministries, HEIs and other 
institutions or stakeholders concerned: 

• “an independent Agency under the auspices of the Ministry” 

• “an independent corporate body under the aegis of the Department of Education” 

• “autonomy of governance arrangements” 

• “Statutes state that the Trust shall carry out its activities independently, 
transparently and objectively.” 

Reports show a great variety of legal acts by competent national/federal/regional 
authorities. This variety of legal acts creates a variety of situations in which agencies 
work, from a very simple legal background to a complex network of legal documents of 
variable importance and impact. Some statutes mix aspects of public and private law. In 
a few cases reports stress the importance of the role and intervention of lawyers. 

The formal legal structure in which an agency works influences not only its form 
of independence but inevitably also the ways in which it may develop and report its 
judgements. Thus, for example, one report noted that: “legal requirements (especially 
because institutions can appeal in court) tend to somewhat dominate, leading to more 
formalistic and legalistic decision documents and perhaps also distracting from a more 
content- and improvement-oriented approach.” 

The coexistence of several (national/regional) agencies within a single or 
overlapping legal/political structures may lead to coordination, division of labour and / 
or rationalization issues. For example, the coexistence of national and regional or 
specialized agencies may mean that more than one set of laws must be considered; one 
agency may be dealing with two sets of laws of two countries; there may be complex 
and unequal partition of functions between national/regional/specialized agencies and 
there may be issues around recognition of the agency as the sole competent quality 
assurance organisation. 

OPERATIONAL INDEPENDENCE 

The assessment of independence is very complex and clearly one of the principles 
most often debated by panels. It is clear that such debates can be difficult both 


within panels and also when comparing the views of the different actors and 
representatives interviewed, since these too are often subject to the ‘internal’ 
contextual (legal, political, historical, social) aspects that can also provide ‘external’ 
panels (whose members have other/different contextual experiences) with difficulties in 
interpretation. 

The reports show that levels of independence are assessed against all possible 
influences, for example independence from the government, from HEIs (in particular 
from applicant institutions), from other stakeholders and from national conflicts 
of interest, and that generally panels are able to deal with the issues in a balanced 
manner, albeit after much discussion. 

FINANCIAL INDEPENDENCE 

Two aspects dominate this area: the extent to which an (any) agency can be 
independent when being dependent on funding either directly or indirectly from 
government funds or from those it is evaluating (the latter may become of increasing 
importance if and when there is a ‘market’ in the quality assurance of European higher 
education), and the necessary resources required to undertake evaluations effectively. 

Panels made various detailed comments within their reports on the types, sources 
and organisation of funding, often (critically) linked to ‘independence’; for example: 
“The current funding model, based on annual subscriptions from HE institutions and 
on a system of annual contracts from the funding councils potentially introduces an 
element of uncertainty into the underlying stability of the agency relating to ‘autonomous 
responsibility for their operations”. 

It is clear from the reports that (at least prior to the onset of the current financial 
crisis) different agencies operated within markedly different financial contexts. The 
balance of funding, and its adequacy from the point of view of independence, is 
viewed very differently from one country (and one panel) to another but it should also 
be remembered that the roles of agencies differ substantially. Whilst no statistical 
correlation has been undertaken in association with this report there is some evidence 
that those agencies with responsibilities for extensive and detailed series of programme 
and personal evaluations, and particularly where there are extensive site visits and or 
panel meetings, require far greater funds than the costs involved in just institutional 
level evaluations. It may also be noted however that those agencies with more limited 
series of programme level evaluations and/or focus on institutional level evaluations 
tend to be able to devote more time and resources to ‘improvement functions’. 

ORGANISATION 

The reports rightly describe and discuss the organisation of agencies in detail with 
regard to their compliance with the Standards and their expectations. Such aspects are 
considered under the following: 

• governance and organisation structures 

• nomination and appointment of executive officers and board members, both of 
have direct impacts on the level of independence (real or apparent) 

• nomination, appointment and training of assessors 

• range of evaluation procedures (and their development and implementation) 

• reporting 

• issues concerned with appeals 

• prevention of conflicts of interest 


Reports inevitably describe a broad range of governance arrangements that are in 
many cases either determined by or largely reflective of the (national/regional) legal 
contexts within which the agencies operate. There is a clear distinction between the 
governing body (or Board) and its (executive) assessment arm with “clear separation of 
powers in its committees as between governance functions (by the Board) and academic 
decision-making”. Whilst there is usually some autonomy of the agency for its internal 
organization, there is always a requirement that the agency’s own internal regulation is 
monitored by the governing body, which may report either directly to government, or 
indirectly through, for example, a published annual report. 

The nomination and appointment of members of governing boards is often formally 
subject to ministerial approval, which can of course lead to questions about the extent 
of the autonomy of the agency. Whilst reports are not quite so blunt, it seems apparent 
that panels make a judgement on the independence standard in the context of the legal 
framework and the extent to which external (ministerial) approval might potentially 
interfere, or provide a more passive formal acceptance of the agency and its work. 

Some formal, governmental nomination regulations are often seen as guarantees 
of authority. Reports show that, in some countries, autonomous organisations are 
nominally headed by the highest authority of the Ministry, (e.g. “Members are officially 
appointed by order of the Minister but in fact nominated by their respective organisations 
and the agency”; “Accountability is to the Council of Ministers (i.e. the Prime Minister, 
not the Minister of Education). The Chair of the Agency enjoys a quasi “ministerial” 
status”.). Other stress however that there is “. . .no formal intervention or influence on the 
appointment of Council members” and in at least one case a consequence of an ENQA 
review has been a change in the law to ensure that there is a formal limitation on a 
minister’s powers of intervention even as formal chair of the governing body. 

Other factors that may impact on independence include the duration of mandates 
and the possibility of their renewal and dismissal conditions (can managers/directors/ 
members be dismissed?) At a practical level reports record a variety of Board 
compositions, with different views about the various possible categories of stakeholders 
and the right balance between them. Boards may include: 

• (no) representatives of Ministries in charge of Education 

• (no) representatives of business, industry, and professional bodies among the 
members of the Council. Exclusion of non academic members 

• (no) representatives of HEIs, of the applicant institutions. 

• (no) student representatives 

Reports include a number of direct recommendations (e.g. “A more balanced makeup of 
the Board including students or representatives from the professional world might make 
the agency less vulnerable to external pressures” (where a Board was mainly constituted 
by representatives from HEI’s and the Government), and proposals for changes in the 
balance of representation in different organs, e.g. : 

• “nationals are not in a majority in the decision-making board” , 

• “Academics are needed, but necessary to include also international and stakeholder 
representatives to improve the accountability of the quality assurance system and the 
ownership of stakeholders” 


The nomination and final appointment of expert members of evaluation commissions, 
and peer reviewers/panel evaluators is also covered extensively in the reports. They 
reflect a variety of views about the definition of criteria and procedures for the 
nomination and final appointment of experts, sometimes clearly noting unusual (but 
perhaps legally required) arrangements, e.g. “the experts and panels are confirmed by 
another and external [to the agency] body” 

A number of reports note a specific situation relevant to small countries/regions: 
“everybody knows everybody else”, and calls for the appointment of experts drawn from 
other regions or even an international pool. There may however be language difficulties 
in some such cases. 

The reports show diverse patterns in the origins of experts, and a number also 
indicate an increasing trend to draw experts from wider sources. Panels range for 
example between all international, and/or all from other regions, to all national, 
with one report noting that, “. . .for increasing objectivity, the Agency uses almost 
exclusively foreign evaluators.” More common, however, was the greater diversification 
in evaluators from exclusively ‘academic’ to the (greater) inclusion of ‘stakeholder 
representatives (including from the labour market) and, of course, students. 

To avoid duplication, organisational aspects of the range of evaluation procedures 
(and their development and implementation), reporting, and issues concerned with 
appeals, are considered in the next (Activities) section. 

PREVENTION OF CONFLICTS OF INTERESTS 

Whilst one report noted that that were “A wealth of technical rules aimed at 
guaranteeing that experts have no vested interest in the HEI or programme/project 
they evaluate”, a wider range of reports noted some limitations to the extent to 
which conflicts of interest were dealt with effectively (if at all). Generally, however, 
experts working on evaluation panels declare in writing that they are independent/ 
sign a “conflict of interests” document, and/or have to subscribe to a code of ethics. 

It is usually the case that the institution under evaluation may object to an expert’s 
nomination on grounds of prejudice, and in a few cases there is some formal 
monitoring, for example where the panel can only officially start its work once its 
independence has been checked by the Higher Education Recognition Commission. 
Interestingly in the latter case the report noted that in about 25 percent of the panels 
some form of incompatibility or dependency was identified (or could not be completely 
ruled out). 

An important feature concerning consistency between reports is the way in which 
panels reported their reservations, and the extent to which these came with or without 
associated recommendations. Thus, for example, 

• “Although the agency is fully compliant with the standard on Independence, the panel 
feels that the present situation could lead to a kind of isolation from society.” 

• “call on [ the agency] to rethink the current structure in terms of the political 
independence .. and if necessary, to reform it no later than the next evaluation in 
approximately five years. This should be the central subject of the next evaluation”. 

• no challenge of practical independence but insistence on symbolic aspects : 

“changing the balance of membership would provide greater symbolic reassurance” 

• “specific recommendations on the future development, funding and governance 
would allay any lingering concerns about independence.” 


Two general conclusions that can be summarised from the ways in which panels have 
applied this standard: 

• should an overall judgment on independence consider the legislative/political 
context in which the agency is working as a form of mitigating circumstances 
where these prevent the agency from fulfilling all aspects of the standard and 
guidelines? Some panels clearly do, whilst others appear to expect a stricter 
interpretation irrespective of the legal context, although both types can offer 
recommendations. 

- ENQA and its partners may wish to consider how greater consistency might 
be achieved, either/both through revision of the Standard’s guidelines or/and 
specific training / guidance for panel members on this matter. 

• The inclusion of recommendations does not always appear consistent. In all 
peer review there is, however, an element of ‘connoisseur’ judgement which 
is understood by specialists but may be less obvious to general stakeholders; a 
similar final judgement can be reached in different ways and for different reasons 
with recommendations being offered where panels have reservations that can or 
should be addressed. 

- ENQA and its partners may wish to consider providing panels with advice on 
how and when to offer recommendations in a more consistent and transparent 
way. 

ACTIVITIES 

Four standards may be considered under this theme: 

• 3.1 Use of external quality assurance procedures for higher education 

• 3.3 Activities 

• 3.7 External quality assurance criteria and processes used by the agencies 

• 3.8 Accountability procedures. 

Standard 3.1 Use of external quality assurance procedures for higher education 
The standard states that “The external quality assurance of agencies should take into 
account the presence and effectiveness of the external assurance processes described 
in Part 2 of the European Standards and Guidelines”. The guidelines add that “The 
standards for external quality assurance contained in Part 2 provide a valuable basis for 
the external quality assessment process ...” and that “The standards for external quality 
assurance should together with the standards for external quality assurance agencies 
constitute the basis for professional and credible external quality assurance of higher 
education institutions”. 

In terms of the overall assessment against Standard 3.1, a third of reports concluded 
with a judgement other than full compliance. However, the reasons for these are not 
consistent across the reports, with different agencies failing to satisfy the evaluation 
panel about different aspects of the ‘sub set’ of standards (those within ESG 2). 
Standards 2.4 (Processes fit for purpose), 2.6 (Follow-up procedures), 2.3 (Criteria for 
decisions) and 2.5 (Reporting) caused the most problems. 

The standards in Part 2 of the ESG are considered in detail below: 


Standard 2.1 Use of internal quality assurance procedures 

The activities listed under 2.1 are those that agencies are expected to apply when 

evaluating the QA activities of HEIs. Thus, for example, under standard 2.1 (Use 

of internal quality assurance procedures) an agency would be looking at. . . “the 

institution’s own internal policies and procedures. . and whether “. . . institutions are able 

to demonstrate the effectiveness of their own quality assurance processes . . 

Agencies generally undertook this work of evaluating whether institutions and/or 
programmes used internal quality assurance procedures satisfactorily. However it is 
perhaps surprising that 3 agencies received only substantial compliance judgements on 
this important aspect. 

In view of the importance of this aspect the panel’s comments are considered 
here in some detail. In one case the panel noted that “. . . [the agency] provides the 
universities with a de facto package of internal quality assurance mechanisms to a very 
high degree of specificity. This leads to the problem that [the agency] could more or less be 
delivering to the universities the internal quality assurance framework later to be assessed 
by [the agency] itself; in other words the agency had strayed too far into the role of 
coach’ whilst also being the referee’. The panel considered “...the guidelines [are] very 
specific and directive in contrast with the generic approach in the ESG. The risk is that 
implementation of the guidelines could lead to a homogenization of the quality system 
within the universities, and difficulties in later adopting new and innovative systems”; an 
interesting comment that might be read as supportive of the agency’s efforts in assisting 
the transformation of an HEI culture from one of strict compliance with set ‘input’ 
standards towards the ethos of the EHEA and its greater emphasis on interpretation of 
standards within context and an added emphasis on (‘output’) achievements. 

In a second case the agency was found to be partially compliant on the basis of 
inconsistencies within the institutions it was evaluating between policies and actions. 

In a number of cases it was noted that such inconsistencies were a consequence of a 
lag between new policies and a residual ‘old culture’ of practice. The panel did however 
note that the agency was active in seeking to assist HEIs in their development of 
effective internal QA units/procedures; a further example of how the ENQA evaluation 
process could be supportive of an agency’s efforts, perhaps in changing less than 
favourable circumstances. 

In the third case the agency works within a context where the national legislation 
does not provide any specific framework for internal quality assurance. The panel was 
however “glad to note that [the agency] has adopted a pro-active and supportive approach 
by including ENQA standards in its external quality assessment” and that “as confirmed 
.. during the site visit, this has already encouraged reflection and desirable developments 
in institutions”. The panel noted however that whilst a “flexible approach to internal 
quality assurance and the focus on progress towards the establishment of internal systems 
rather than the effectiveness of fully-fledged systems is fully justified” (i.e. the agency was 
taking care with its role as ‘coach’ in light of its subsequent ‘referee’ role) never the 
less, there is no timeframe set which could provide, “. . .an extra incentive to institutions 
to introduce their .. internal quality assurance systems”. This led to the substantial 
compliance judgement. 


Standard 2.2 Development of external quality assurance procedures 
Agencies generally undertook this work satisfactorily. However it is perhaps surprising 
that three agencies received only substantial compliance judgements on this important 
aspect. 

In all three cases panels were concerned with the process and perspectives of the 
development of procedures. In one case the agency, although established for some years 
and without a restrictive legal context, was regarded as taking a too ‘retrospective’ 
approach. In another, where there was a restrictive legal framework, the panel noted 
a continuing over-reliance on ‘input measures’ albeit within a variety of approaches. 
Recommendations were provided in these and other cases where compliance with the 
standard had been recorded but panels still felt there was scope for improvement. 

Standard 2.3 Criteria for decisions 

All agencies undertook this aspect of their functions satisfactorily. 

Standard 2.4 Processes fit for purpose 

Agencies were generally found to have processes that were fit for purpose. However two 
agencies received substantial compliance judgements on this important aspect. 

In one case the panel noted that the agency had ‘imported’ procedures from 
elsewhere that, whilst designed for specific purposes, were perhaps not ideally suited 
for the agency’s own responsibilities. It assessed that the agency was quite capable of 
(redesigning its procedures to meet its specific needs and thus recorded a substantial 
compliance. 

In a second case, whilst the panel was largely content with the work of the agency 
under this standard, it was concerned by a few inconsistencies between procedures and 
also, and more importantly, within methods (particularly concerning the inclusion or 
not of site visits); these led to a judgement of substantial compliance. 

A consistent comment in many reports (and in some cases leading to 
recommendations) was the limited extent to which agencies sought input from 
‘external’ stakeholders, particularly from the professional and/or business sectors, 
when developing their procedures and criteria. Often the processes were designed and 
implemented with introspective HE/legal frameworks being the only considerations in 
mind. 

Standard 2.5 Reporting 

All agencies were found to be working satisfactorily with regard to reporting although 
there were several recommendations; it is clear that some panels have (rightly?) 
interpreted the ‘spirit of the standard’ in so far as it can be applied within the legal 
context in which an agency is working. 

Standard 2.6 Follow-up procedures 

Agencies generally carry out this work satisfactorily. However three agencies received 
only substantial compliance judgements on this aspect. 

There is some superficially apparent inconsistency between panels in reaching a 
judgement on this standard. Various agencies are, for a variety of external reasons, 
constrained in their actions, but panels appear to have been more or less generous in 
the extent to which they have allowed for ‘mitigating circumstances’. Agencies are at 


different stages of maturity and several are affected by changing legislation; in some 
cases panels have made allowances for these factors in applying the ‘spirit’ of the 
standard whilst in others there appears to be less flexibility. What a simple reading of 
the report does not always convey is the extent to which the circumstances were a real 
cause of difficulty or perhaps an excuse for lack of consistent and transparent activity. 
Any peer assessment system will be subject to such variations but it is incumbent on 
ENQA to assist in reducing variability between panels by guidance and training. It 
might be of greater concern if a simplistic use of a panel’s judgements were used for 
subsequent actions and purposes. 

Standard 2.7 Periodic reviews 

Almost all agencies met this standard satisfactorily but two agencies received only a 
substantial compliance judgement. There is evidence that some panels were inclined to 
apply the ‘spirit of the standard’ where agencies were constrained either by the stage of 
development of their work, or the legal constraints they operated within. 

In one case the panel noted that it “[had] not been able to identify the length of the 
cycle for [some] current activities . . The panel also recorded that “it is important to 
notice that the review panel acknowledges that the regular revisions of legislation are 
mostly responsible for this “stop and go” context and that [the agency] is excusable for the 
conclusion below in the sense that it has had to face this legal situation .. they none the 
less returned only a substantially compliant judgement. 

Standard 2.8 System wide analysis 

Agencies generally met this standard satisfactorily. However three agencies received 
substantial compliance judgements on this aspect. 

Substantial compliance resulted in one case from the panel finding that there is no 
analysis and comparison between the different [approaches] [used by the agency]; again 
with the panel making what appears to be a constructive recommendations within the 
context in which the agency works. 

In another case the panel noted that whilst the agency “provides annual reports with 
aggregate information about its evaluation, certification and accreditation activities. . . 
[had] also begun to provide a limited number of cross-sector reports on areas such as 
Library provision [and] proposes to expand its capacity to undertake further system-wide 
analyses ” .. “[it] believed that there was more work to be done in this area of enhancement 
activity”. 

It is clear that panels found a number of areas of serious concern to them that 
resulted in judgments other than full compliance. In all of these and also in several 
cases where compliance was confirmed panels offered recommendations that were 
designed to assist the agency directly in its activities and also in some cases intended to 
assist the agency with influencing legal ‘pressures’ that constrained its work within the 
spirit of the ESG. 

A general conclusion can be summarised from the ways in which panels have applied 
this standard: 

• the lack of any consistent reasons why 6 agencies failed to gain full compliance 
indicates that the standard is appropriate. The reports also indicate that it is an 
essential one and further that agencies are working to overcome any shortcomings 
which are generally dependent on legal/cultural contexts or simply the relative 


‘age’ of the agency (it simply had not yet had time (or in some cases needed) to 
develop and implement the full panoply of tools and procedures expected). 

Standard 3.3 Activities 

The standard states that “Agencies should undertake external quality assurance activities 
(at institutional level) on a regular basis” with the guideline adding that “These may 
involve evaluation, review, audit, assessment, accreditation, or similar activities and 
should be the core functions of the agency.” It is of note that the ‘Activities’ standard does 
not refer to any ‘improvement/enhancement’ activities although many agencies claim 
to undertake these either with regard to improvement of QA procedures themselves 
(although this aspect may be regarded as implicit in Standard 8: Accountability 
procedures) or through contributing to improvements in the ‘subject’ under evaluation 
(a programme or institution). 

The Activities standard was one that agencies under review managed to deal with 
very successfully; all undertook external quality assurance on a regular basis, although 
several reports note that the activities undertaken by the agency had changed and 
would continue to change, within structured (although sometimes quite ambitious) 
strategic and operational plans. 

Before considering the range of those activities it might be reasonable to ask ‘why, if 
it seems so straightforward for agencies, is this a standard?’ The answer may lie within 
the context of the stage in development of national, regional and other agencies across 
the EHEA when the ESG were first proposed. There were legitimate concerns about the 
potential risk of the development of transient, for-profit QA agencies - the evaluation/ 
accreditation equivalent of ‘degree mills’. It may be a success of the ESG that such a 
scenario has not (yet) arisen within the ever expanding number of agencies across an 
increasingly broad EHEA. 

The activities undertaken by the evaluated agencies vary considerably in scale 
(gathering information on between 5 and 1500 HEIs!) and scope. The activities of 
some agencies are limited essentially to the programme level whilst others are also 
involved at the institutional and/or specialist ‘unit’ level, and/or with the evaluation of 
staff as well. Evaluation of staff may be in a direct manner, evaluating individuals (with 
consequences for their careers, salaries etc) or indirectly, but still rigorously, through 
an evaluation of the staff associated with a specific programme under scrutiny. Strict ex 
ante staffing requirements have, in some cases, required the development of a data base 
to monitor, and eliminate, the phenomenon of the ‘flying professor’ - where esteemed 
individuals are listed as contributing to an excessive number of programmes often over 
a range of different institutions. 

There is increasing evidence of a general pattern of development from ‘universal’ 
programme evaluation towards assessment at the institutional level, with or without 
some evidence being gathered at programme level as well. It is clear from the reports 
however that not only do different agencies fulfil different ranges of activities but also 
that the subjects they evaluate have different governance arrangements; some HEIs 
have a strong centralising Rectorate whilst others HEIs are essentially an association 
of largely independent units and faculties. It will be interesting to see whether the 
application of the more regular external evaluation on the basis of the ESG will tend 
to encourage institutions towards more centralised co-ordination and consistency of 
internal QA within and across the whole institution. Linked closely with this is the 


legal context and the nature of the awarding body; whether awards are made in the 
name of the institution or the state. 

One aspect that provided for discussions within some evaluation panels was the 
‘match’ or not between the agency’s mission statement and its activities. The ‘mission 
statement’ standard (ESG 3.5) refers to having a “. . .publicly available statement... 
translated into a clear policy and management plan”, but nothing about specific 
current activities. For example, it is possible for an agency to claim within the 
mission statement that its main aim and activities are to contribute to improvement/ 
enhancement, yet in the early years the activities will almost certainly be concerned 
with building up the data and expertise upon which such an enhancement function 
might in future be delivered. In such a case the agency has a mission statement 
(compliant with 3.5) and it has regular activities (compliant with 3.3) but they are not 
(yet) directly linked. 

A general conclusion can be summarised from the ways in which panels have applied 
this standard: 

• agencies are undertaking the QA activities expected of them within the concept 
of QA within the EHEA. However, there remains a question about whether and 
the extent to which agencies might/should also be engaged in an improvement / 
enhancement agenda. If so, should they be restricted to improvement of QA 
instruments and procedures or should they also have a role in development of / 
assessment of improvement /enhancement of academic pedagogical aspects? 

Standard 3.7 External quality assurance criteria and processes used by the agencies 
The standard states that “The processes, criteria and procedures used by agencies should 
be pre-defined and publicly available. These processes would normally be expected to 
include: 

• a self assessment or equivalent procedure by the subject of the quality assurance 
process 

• an external assessment by a group of experts, including as appropriate (a) student 
member(s), and site visits to be decided by the agency 

• publication of a report, including any decisions, recommendations or other formal 
outcomes 

• a follow-up procedure to review actions taken by the subject of the quality assurance 
process in the light of any recommendations contained in the report.” 

The guidelines add that “Agencies may develop and use other processes and procedures for 
particular purposes. Agencies should pay careful attention to their declared principles at 
all times. . .” 

Only 60% of the evaluations covered by this report resulted in a (full) compliance 
judgement with Standard 3.7; the others were all partially or substantially compliant. 

It is interesting to note that the ratio between full compliance and partial/substantial 
changed over the period covered by the reviews; in the 2009 reports there are more 
substantial than full compliance judgements, whereas in 2007 and 2008 there were 
more full compliance judgements. Is it the case that agencies did not seem to have 
learnt from the shortcomings of others, or that the panels were getting ‘tougher’ in 
their expectations, or that the types of agencies submitting themselves in 2009 were 
different from those in the previous years? 


Examination of the reasons for partial/substantial, as opposed to full compliance 
indicates that the following were seen as shortcomings: 

• lack of student members in the evaluation teams (several) 

• lack of an internal appeals procedure (several) 

• lack of follow-up procedures (several) 

• lack of site visit in some activities 

• variation in consistency of an agency’s panels (although was already recognised 
and being addressed) 

• limited stakeholder involvement 

• over emphasis on ‘input’ criteria/ measures in evaluation procedures 

• issues concerned with publication of the report (several different aspects) 

Of the above list some are essentially structural (working within quite tightly defined 
legal frameworks), some are developmental (agencies have just not been running long 
enough to need e.g. follow-up procedures), and some are size related. 

However, some are of a more fundamental nature - e.g. the lack of an appeals 
procedure or variations in consistency between an agency’s panels working in the same 
area. It is interesting to note that in almost all of these cases the agency had within 
its self evaluation process identified its shortcomings and had started to take steps to 
remedy them; the panel’s judgements were based on the fact that there was not yet any 
evidence of change. 

The structural issues are, in some cases, harder to deal with since some are bound 
up in national legislation and/or the nature of the detail of evaluation procedures. 
Examples here would include the legally bound programme standards often prepared 
at a time and in a context predating the more recent shift towards recognising the 
importance of learning outcomes; in such cases change is underway or imminent 
but at a pace set by change of national legislation. Similarly, the detail of programme 
evaluation for some agencies includes detailed scrutiny of individuals involved in 
teaching on the programme and an assessment of their research. In such cases it 
is argued that full publication of data on individuals would not (and should not) 
be permissible - the guideline “publication of a report, including any decisions, 
recommendations or other formal outcomes” is thus seen to become difficult to fulfil in 
full. There is some evidence in the reports of differences in interpretation of this aspect 
- some panels appear to have read the guidelines within the local legal context whilst 
others have taken a more ‘literal’ stance. 

The reports clearly demonstrate that most agencies have noted weaknesses in this 
area during the development of their SER and that have, by the time of the site visit, 
been active in starting to develop/implement changes. 

General conclusions can be summarised from the ways in which panels have applied 
this standard: 

• this is an essential criterion in the evaluation of agencies, but 

• many agencies are constrained in which they seek to fulfil it by local legal 
requirements/limitations 

• panels vary in the extent to which they are prepared to include local contexts 
as ‘mitigating circumstances’ when reaching their judgement on (the level of) 
compliance. 


Standard 3.8 Accountability procedures (in the context of activities) 

The standard states that “Agencies should have in place procedures for their own 
accountability”, with guidelines covering a published policy for the agency’s internal 
QA, various forms of documentation to support and demonstrate this and a mandatory 
cyclical external review. An interesting omission in the guidelines is any explicit 
reference to the agency demonstrating accountability to the interests of its key 
stakeholder(s); the standard and its application seem to consider feedback merely from 
the ‘subjects’ of the evaluation not whether, for example, the wider public interest has 
been met by the evaluations. An almost exclusive emphasis on accountability through 
the refining of procedures may be regarded by some as somewhat introspective, and 
some panels introduced the wider expectation (since it was not explicitly excluded 
either). 

Approximately 25% of the agencies evaluated failed to get a ‘full compliance’ 
judgement on their accountability procedures. Reasons for partial/substantial 
compliance included: 

• lack of a formal and systematic mechanism for feedback to HEIs 

• only limited application of ‘no conflicts of interest’ policy (several) 

• lack of ‘wider’ accountability (several) 

• lack of formal timetable for the frequency of mandatory external reviews 

• limited training of peer review panels 

Several panels make recommendations that suggest that the agency considers the 
development of a Code of ethics to address the ‘no conflicts of interest’ expectations, 
and in particular that this should distinguish clearly between those involved in internal 
and external quality assurance, and their respective roles. 

One panel took the view that “.. [the agency’s] accountability is currently focussed on 
HEIs. . its accountability towards society at large is not sufficiently provided for. . .“. The 
report also notes that “The agency is fully aware of this and has signalled its intention to 
address the issue. . .“ yet it still received a substantial compliance for this standard. 

Appeals mechanisms are relatively straightforward to put in place and it is clear from 
the reports that, if such mechanisms were currently lacking, this had been recognised 
by the agency and they were likely to be in place in the near future. Perhaps more 
serious are the few references to limited or inconsistent training of the panels of peer 
evaluators. This issue is critical and is being addressed at more strategic and practical 
levels by ENQA. 

Appeals mechanisms 

Only a minority of agencies mention an existing formal appeals procedure. The 
internal quality assurance system may involve procedures for processing remarks 
and complaints: “All remarks, problems or complaints can be reported to staff members 
or to the Unit’s coordinator. They will try to mediate any problems. There is always the 
possibility to enter a formal procedure of arbitration”. Where a formal appeals system 
exists, only few vague indications, if any, are given on cases or numbers of appeals, and 
none on specific appeals issues and decisions. Some panels suggest that this is because 
the review was too early and that such a procedure is still under development. 

Different possible processes may be detected under the theme of “appeals”: for 
example, there may be an extension of the usual process that goes beyond just the 


correction of factual errors, to contest proposed judgments before the finalization 
of reports or decisions: “The university colleges through their programmes have two 
opportunities to comment on their assessment reports. “It is the sole competency of the 
expert panels to decide whether these comments are included in the final assessment report 
or not.” 

There may also be different types and levels of appeals: 

• an internal appeals to a formal committee within the agency, or “an independent 
Appeals Board” without precision about the scope of this independence 

• an external appeal (at a Court) 

A few agencies mention that they are in the process of creating such a procedure: “it 
emerges that for the HEIs accredited by the agencies, the possibility of opening a grievance 
process exists” 

“At the request of the institutions the agency is now in the process of establishing a 
formal appeal procedure. In case of a conflict an independent appeal panel will in the 
future decide an outcome.” 

Where they exist, these appeals procedures are often stipulated by law (as an 
application of the general national legislation on public administrative decisions). 

A general conclusion can be summarised from the ways in which panels have applied 
this standard: 

• the standard appears to be ambiguous as regards the purpose/audience of 
accountability and this leads to difficulties for panels 

• an agency should certainly have procedures to ensure internal monitoring and 
accountability of its activities; is it also required to undertake and provide public 
accountability for the external impact of its activities? 

SUSTAINABILITY 

The group of standards under ‘sustainability’ deals with the development, consolidation 
and improvement /enhancement of agencies’ activities and looks at whether the 
existing levels of resource and accountability will be sufficient to ensure continuation of 
the agency and its activities in the future. Two standards may be considered under this 
theme: 

• 3.4. Resources 

• 3.8. Accountability procedures 

Standard 3.4 Resources 

The standard states that; “Agencies should have adequate and proportionate resources, 
both human and financial, to enable them to organise and run their external quality 
assurance processes) in and effective and efficient manner, with appropriate provision for 
the development of their processes and procedures.” 

The sections of the reports that deal with resources are predominantly descriptive. 

In most cases they take for granted agencies’ statements or reflect the panels’ 
impressions. However some give various insights and more in-depth analysis that could 
be considered further in any review of the ESG. 

SERs usually, but not always, express satisfaction about the adequacy of their 
resources, including budget, internal staff numbers, office space, equipment and 
material resources and appropriate levels of reference materials, with reports indicating 


that the panels usually confirmed these views, sometimes with additional observations, 
e.g.: “The fact that it [the agency] considers the resources available to be sufficient for its 
current tasks clearly demonstrates that it organises and performs its work in an efficient 
and effective manner. A mid-term financial perspective would certainly be desirable”. 

However, panels also express concern about the impact of the evolution of 
missions on agency resources, workloads and some other specific cost factors. They 
try to anticipate factors that may put pressure on resources and generate additional 
expenditure: 

• “it may need to extend its range of staff resources as it maintains current delivery and 
extends its activities to more universities. 

• “additional human resources would be needed “if the Unit wishes to achieve all 
its ambitions (monitoring and contributing to the development of a new system, 
performing wider-ranging substantive analyses ;)” 

• “the review panel considers the resources available adequate for the time being. 
However, it seems not unlikely that the present level of provision will no longer be 
adequate sometime in the near future”. 

Specific missions (e.g. teaching and research staff assessment), permanent or ad hoc, 
imposed on agencies by law and/or by their authorities have an influence on their costs, 
their allocation of resources and staff and the definition of their priorities, planning 
and operations. Some reports thus mention discrepancies between legal assignments 
and level of public funding, e.g.: “the government does not contribute towards the funding 
even though the quality assurance processes it undertakes are mandatory by law”. 

Whilst there are no consistent patterns of underfunding of particular areas/activities, 
reports do include comments about insufficient resourcing in several areas, mostly 
concerned with functions that might be regarded as not being of direct or primary 
concern, for example: 

public relations, events and seminars, staff development, international networking 
and analysis of relevant processes. Two other areas that are mentioned in several 
reports are, however, a reflection of changing expectations of quality assurance that 
might now be regarded as primary functions/requirements and yet were not considered 
when budgets were established some years ago; these include the costs of: 

• the inclusion of professional, students and international experts in future review 
activities or research projects 

• assisting with the development of internal quality assurance systems within 
institutions. 

Some reports reflect the problems faced by agencies as they consider how to adapt 
their work pattern to the evolution of their missions, with reports generally offering 
supportive comments, e.g.: 

• “a changed mix of professional and administrative skills will be required, particularly 
in policy development and support of capacity building for quality enhancement in 
the higher education sector”. 

• “the Agency has now reached a point of maturity in its development where the 
appointment of a senior professional manager to support the Director in taking 
oversight of some of the key processes might be appropriate”. 

• “The panel sees important advantages of a long term professional secretary general 
position”. 


Despite their potential significance, fees are not reported on frequently but concern 
is demonstrated by some panels, for example: “It is the Panel’s opinion that (the low fee 
to the members of the expert teams and the students that participate in the accreditation 
processes) constitutes a risk for the Agency, since the best qualified experts may not be 
attracted to participate in the accreditation procedures. At the international level, low 
fees will be an important obstacle to the recruitment of foreign experts for the assessment 
teams”. 

Only a few SERs and reports give a detailed description of funding and expenditure, 
including their evolution in time by activities, type of expenditure and sources of 
funding, unitary costs by operation type, basic components of costs, and evaluation of 
additional costs due to extra load imposed by ministries or to be forecasted because of 
expansion of missions. These examples might be used for more in-depth comparative 
questioning and information in training sessions. 

A general conclusion can be summarised from the ways in which panels have applied 
this standard: 

• External quality assurance, particularly at programme level, can be costly 
especially where site visits are a routine expectation; but with substantial 
political backing for the Bologna Process and with quality assurance as a central 
pillar of that process, agencies until now appear to have been generally quite 
well resourced. It will be interesting to see whether this remains the case as a 
consequence of the global economic downturn and it may be that agencies will 
need to find more cost efficient approaches and procedures in future. 

Standard 3.8 Accountability (in the context of sustainability) 

The standard states that, “Agencies should have in place procedures for their own 
accountability.” 

Whilst most agencies were able to demonstrate that they met this criterion, it is clear 
from the reports that this standard, as written, led to discussions and difficulties for the 
evaluating panels. The standard itself appears to be unclear as to whether it refers to 
the agency having procedures to ensure that it can demonstrate accountability to itself 
- or to external audiences, or indeed whether the former should, in effect, satisfy the 
latter. The associated guidelines provide a series of exemplar activities but the ‘audience’ 
for the evidence provided remains unclear. 

It is, of course, more straightforward for an agency to engage in a series of 
activities that can satisfy itself that it is performing satisfactorily: e.g. sampling to 
see if its procedures and processes run smoothly and on time, and all agencies are 
involved in such activities. From an external viewpoint however another question 
is “do the processes and procedures generate worthwhile outcomes/consequences 
for stakeholders?” It is clear from the reports that few agencies monitor their 
impact/value added’ for others. 

As a preamble to accountability a few agencies present their published quality 
assurance policy as a general cultural systemic issue related to all aspects of the 
agency’s activity: “to create a common evaluation culture within the entire organization”. 
Basic action principles are then expressed in a variety of ways: mission statements, 
declarations of principles, reference to recognized international practice in the area, 
etc. seeking to demonstrate (more or less explicitly) claims such as: 


• “independence, objectivity, transparency, efficiency, distance and fairness” 

• “consistency and equal treatment, openness and clear communication externally” 

• “rigour, quality and objectiveness in its processes, independence of stakeholders” 

• “ work [that] is carried out so that external parties can benefit from process and 
result” 

Agencies are effective in providing internal quality assurance systems that are, in most 
cases, defined in separate, publicly-available documents. In a few cases most, if not 
all, elements of such policies exist, but without being summarized in one document; 
agencies know what they are doing and why, and seek to improve their effectiveness. 
Most have a view of fostering continuous improvement in the management system and 
their key processes, with for example: 

• “preparation and monitoring of a Strategic Plan for 2008-2010, supported by an 
annual work plan”, 

• “The quality system includes procedures and guidelines for the quality work in the 
operating units as well as annual quality assessments and external feedback.” 

Much thought and time is spent on internal (bureaucratic) planning, reporting 
and checking and agencies publish and use many quality assurance documents: 
quality handbooks, internal quality system manuals, Codes of ethics, Codes of Good 
Practices, Codes of Confidentiality, etc, some of which are beyond what might be 
legally required but which agencies regard as improvement/enhancement or quality 
assurance procedures and processes. Panels have on occasions recommended that, 
where appropriate, documents and processes be more formalised, for example: “In 
general, the internal quality assurance still relies for a substantial part on relatively ‘soft’ 
instruments. Given the scale and the number of key persons and parties involved, this is 
feasible. However, the panel would welcome a more systematic approach: periodic reviews 
of the processes, an annual action plan et cetera”. 

The reports do however indicate the beginnings of a more outward perspective in 
terms of accountability. The choice of an appropriate periodicity for reviewing activities 
and processes is a growing concern. Some agencies now consider that they need not 
only to have ad hoc occasional feedback but also more of “a regular pattern of self-review 
and a readiness to obtain information about the effectiveness and impact”. However, the 
frequency of mandatory external reviews of an agency is not always defined in official 
documents. The most frequent is a five year external review cycle which corresponds 
to ENQA and ESG requirements. Interestingly the Guidelines, which establish the 
desirability of the 5 year external review cycle refers only to “the agency’s activities”, not 
any impact of those activities. 

Agencies (and to some extent their sponsors) see accountability as an opportunity 
to improve their (internal) management by bringing consistency between ESG/EHEA 
requirements and their own legal requirements. Thus, for example, one report noted 
that whilst “periodic external evaluations are not prescribed by the law, [they are] 
considered mandatory by the agency itself, within a European perspective.” Where such 
an external review cycle is not yet effective, it is at least mentioned as an intention, 
or a beginning of implementation; one panel observed, for example, that [such] “. . . 
activities are not part of a formally agreed cyclical pattern of review on a periodic basis” 
whilst clearly noting that the agency was seeking to improve its accountability. It is also 


clear from review reports that agencies’ annual reports also contribute an element of 
continuity, feed-back and periodicity in internal QA. 

Agencies sometimes submit themselves to external reviews which award labels 
or quality certifications in addition to the prescribed mandatory cyclical review, for 
example: “The review will also be informed by a parallel process, namely the achievement 
of a quality mark. Excellence through People, which is the national standard for the 
development of human resources”. Clearly the value/impact of such internally-focused 
activities will be perceived in different ways by different stakeholders, as would a 
number of the other examples of documentation suggested within the Guidelines. 

Nevertheless, with regard to ESG 3.8. Guideline 2 (Agencies are expected to 
demonstrate they have in place internal quality assurance procedures such as an 
internal feedback mechanism, an internal reflection mechanism, an external feedback 
mechanism in order to inform and underpin its own development and improvement) the 
reports demonstrate that a great variety of mechanisms exist, which are more or less 
documented and systematized, and which are reviewed with different periodicities. 
They can/should be considered from the point of view of identifying and achieving the 
right balance between internal and external quality assurance processes. 

Methods referred to in the reports include: 

• suggestion forms and boxes, personal interviews, satisfaction surveys, periodic 
meetings with staff, meetings with the board of directors, electronic surveys, 
questionnaires, from institutions and from evaluation teams 

• mechanisms of internal reflection: regular staff meetings, briefings of 
the management and administration, workshops, reviews of structures of 
departments, procedures, job descriptions on the basis of feedback, 

• annual conferences of experts for sharing experience and good practice. 

Some agencies have a published document entitled “Internal Quality Assurance Policy”, 
or “Internal Quality Assurance and Accountability Policy” which sets out the general 
policy framework and structures through which the relevant authority assures the 
quality of the organization itself and demonstrates its accountability. Analysis and 
comments by review panels shows a concern for a systematic approach to IQA: 

• “the internal quality assurance system is underdeveloped” 

• “the agency has not yet developed a formally differentiated internal quality assurance 
system, but is applying several external feedback mechanisms and processes” 

• “the openness of documentation and procedures provides additional assurance of 
quality in itself.” 

Panels clearly expect at least a description of internal quality assurance systems, 
characterized by various levels of formalism or informality, but there is an issue here 
of the consistency of all these devices and methods, designing and using them in a 
coherent and priority-conscious way in order to prevent stress and waste which could be 
brought on by “over evaluating” and “over strategizing.” One report noted for example 
that there was: “No evidence of any established policy for own internal quality assurance 
with regard to its overall effectiveness as an organization, or clear evidence that feedback 
from the different methods was assembled and considered in a holistic and systematic 
manner”. 


Panels do however usually recognize the need to allow time for agencies to consider 
all the organisational implications of such a quality policy, e.g.: “The agency is only at 
the early stage of defining a systematic approach to informing and reviewing the quality of 
its service.” 

The development of structured feedback mechanisms appear in rare cases to be 
prevented by over reactive attitudes or the lack of a sufficiently shared understanding 
of the usefulness of consistent feedback processes, and on rather infrequent occasions 
there were comments such as: 

• “not always open to criticism, considering criticism sometimes as an attack on itself, 
instead of making use of it in improving operations”, 

• “(there is) not yet a culture of continuous improvement. Staff are still in a process 
of understanding of how the quality management procedures should be embedded 
throughout the work processes.” 

• “The panel is of the opinion that the agency would find it useful to “reflect on 
the wider expectations of accountability, and identify the specific roles and 
responsibilities of each stage in its procedures”. 

Clearly for some the right balance is still to be found between not just the collection but 
also reflective use of external feedback. 

External accountability and agencies’ reputations 

ESG 3.8 is not explicit about the role and place of stakeholders with regard to 
accountability processes, either as contributors to the accountability process or, and 
perhaps more importantly for some, as those to whom some measure of accountability 
should be addressed. Whilst most utilise external stakeholders in their processes for 
internal accountability (are our procedures working well enough?) it was less common 
to find reference to external accountability in the sense of ‘are the agency’s findings 
and reports useful to. . .?’ Some panels were clearly content to monitor the ‘internally 
focussed’ nature of the accountability standard whilst for others there was a clear 
need that even if the standard itself was not explicit then an agency should consider 
its accountability to the ‘external’ world. Without such consideration an agency’s 
‘reputation’ could quickly come under very direct (and perhaps politically motivated) 
‘threat’. 

In terms of the participation of the various stakeholders, the reports demonstrate 
that agencies are concerned about their wider reputation beyond the agency itself, e.g.: 

• “the academic community feels actively involved and taken seriously as a partner”. 

• “the agency’s accountability is currently focused on HEIs”. 

• “widely accepted by the higher education community, its accountability towards 
society at large is not sufficiently provided for, not yet properly acknowledged in 
media”. 

• “the agency should be more proactive in communicating with a broader public.” 

For some agencies, but perhaps fewer than might be desirable (i.e. all!) the issue does 
incorporate not only feedback mechanisms but also the introduction of different types 
of external stakeholders to positions in different agency committees, consultations and 
other mechanisms, e.g.: 


• “Accountability is also determined by the way in which the various stake-holders are 
involved”. 

• “The complex stakeholder context within which operates makes necessary a similarly 
complex accountability system” 

• The Panel confirms that the agency delivers substantial accountability to key 
stakeholders through a variety of mechanisms. “ 

In an attempt to include greater involvement by and demonstrate accountability to 
external stakeholders some agencies have established an advisory board, with various 
functions, e.g: 

• “In order to involve more external stakeholders, an Advisory Board consisting only of 
external stakeholders is currently being established”. 

• giving stakeholders a substantial representation in an advisory board would be 
beneficial in terms of accountability”. 

• “The newly established Advisory Board is a first step but the review panel feels that 
the agency could and should consider being more open to external viewpoints”. 

• . .appropriate to ask the advisory board to take on a more substantial role, whilst 
remembering that its title indicates the status of its conclusions.” 

Mechanisms to ensure the quality of subcontracted services and materials 
This is a concern only for a minority of agencies which make use of sub-contracted 
work. Only two agencies mention that they have procedures and mechanisms for 
the evaluation and monitoring of suppliers and subcontractors, ensuring the quality 
of their activities, “to ensure that all work produced externally by subcontractors is of 
an acceptable standard and fit for purpose”. A few other mention that this issue is not 
relevant for them, because they make very little or no use of sub-contracted work. 

A general conclusion can be summarised from the ways in which panels have applied 
this standard: 

• Agencies generally managed to meet the expectations of accountability reasonably 
well with two thirds receiving a full compliance judgement and, except one, the 
others being substantially compliant with the standard. 

• The standard appears to have presented some panels with problems; in its 
guidelines it appears to focus primarily on internal accountability but for many 
there is also a need for accountability to include external aspects - is an agency 
monitoring whether it is actually providing information that stakeholders use and 
consider useful, and is that having any significant impact? 


Annex 2: Reports on agency reviews 
conducted between September 2005 
and December 2009 (with date 
of publication of report on ENQA 
website) 


• Agencia para la Calidad del Sistema Universitario de Castilla y Leon (ACSUCYL) 
December 2009 (published February 2010) 

• Stiftung Evaluationsagentur Baden-Wuerttemberg (EVALAG) September 2009 
(published February 2010) 

• Agency for Quality Assurance in the Galician University System (ACSUG) July 
2009 (published November 2009) 

• Flemish Interuniversity Council Quality Assurance Unit (VLIR - QAU)May 2009 
(published September 2009) 

• Russian National Accreditation Agency (NAA) November 2008 (published 
September 2009) 

• Institutional Evaluation Programme of the European University Association (EUA- 
IEP) April 2009 (published September 2009) 

• Accreditation Agency for Study Programmes in Health and Social Sciences 
(AHPGS) March 2009 (published June 2009) 

• Irish Universities Quality Board (IUQB) September 2008 (published June 2009) 

• Commission des Titres d’Ingenieur (CTI) April 2009 (published June 2009). 

• Romanian Agency for Quality Assurance in Higher Education (ARACIS) May 
2009 (published June 2009) 

• Agency for Quality Assurance in Higher Education and Research of Andalusia 
(AGAE) January 2009 (published April 2009) 

• State Accreditation Committee of Poland (PKA) December 2008 (published 
March 2009) 

• Accreditation, Certification and Quality Assurance Institute (ACQUIN) March 
2006 (published March 2009)* 

• National Qualifications Authority of Ireland (NQAI) September 2007 (published 
March 2009) 

• Agentur fur Qualitatssicherung durch Akkreditierung von Studiengangen (AQAS) 
February 2007 (published March 2009)* 

• Flemish Council of university colleges (VLHORA) September 2008 (published 
March 2009) 

• Quality Assurance Agency for Higher Education, United Kingdom (QAA, UK) July 
2008 (published March 2009) 

• National Evaluation and Accreditation Agency (NEAA) July 2008 (published 
September 2008) 

• Central Evaluation and Accreditation Agency Hannover (ZEvA) December 2005 
(published September 2008) 


• Hungarian Accreditation Committee (HAC) June 2008 (published September 
2008) 

• German Accreditation Council (GAC) April 2008 (published September 2008) 

• Norwegian National Quality Assurance Agency (NOKUT) February 2008 
(published June 2008) 

• Austrian FH Council (FHR) November 2007 (published June 2008) 

• Austrian Accreditation Council (AAC) September 2007 (published June 2008) 

• Austrian Agency for Quality Assurance (AQA) November 2007 (published June 
2008) 

• Accreditation Organization of The Netherlands and Flanders (NVAO) September 
2007 (published January 2008) 

• Foundation for International Business Administration Accreditation (FIBAA) May 

2006 (published January 2008)* 

• Catalan University Quality Assurance Agency (AQU) August 2007 (published 
September 2007) 

• National Agency for Quality Assessment and Accreditation of Spain (ANECA)June 

2007 (published September 2007) 

• Center for Accreditation and Quality Assurance of the Swiss universities (OAQ) 
August 2006 (published July 2007) 

• Accreditation Agency for Study Programmes in Engineering, Informatics, Natural 
Sciences and and Mathematics (ASIIN) February 2006 (published July 2007)* 

• Swedish National Agency for Higher Education (HsV) December 2005 (published 
May 2007) 

• Danish Evaluation Institute (EVA) September 2005 (published April 2007)* 

• Higher Education Training and Awards Council (HETAC) May 2006 (published 
April 2007) 


* indicates reports that do not make specific reference to assessment against the ESG 
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